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INSPECTOR  GENERAL 

DEPARTMENT  OF  DEFENSE 
400  ARMY  NAVY  DRtVE 
ARLINGTON,  VIRGINIA  22202-2884 


June  14,  1996 

MEMORANDUM  FOR  UNDER  SECRETARY  OF  DEFENSE  FOR  ACQUISITION 

AND  TECHNOLOGY 

SUBJECT:  Audit  Report  on  the  Audit  of  Risk  Management  Programs  for  Defense 
Acquisition  Systems  (Report  No.  96-162) 


We  are  providing  this  audit  report  for  information  and  use.  We  considered 
management  comments  on  a  draft  of  this  report  in  preparing  the  final  report. 

Comments  on  the  draft  of  this  report  conformed  to  the  requirements  of  DoD 
Directive  7650.3  and  left  no  unresolved  issues.  Therefore,  no  additional  comments  are 
required. 

We  appreciate  the  courtesies  extended  to  the  audit  staff.  Questions  on  the  audit 
should  be  directed  to  Mr.  John  Meling,  Audit  Program  Director,  at  (703)  604-9091 
(DSN  664-9091)  or  Mr.  David  Wyte,  Audit  Project  Manager,  at  (703)  604-9027 
(DSN  664-9027).  See  Appendix  G  for  the  report  distribution.  The  audit  team 
members  are  listed  inside  the  back  cover. 


Robert  J.  Lieberman 
Assistant  Inspector  General 
For  Auditing 


Office  of  the  Inspector  General,  DoD 


Report  No.  96-l<>2  June  14, 1996 

(Project  No.  5AE-0062) 


Risk  Management  Programs 
for  Defense  Acquisition  Systems 

Executive  Summary 


Introduction.  All  acquisition  programs  are  subject  to  risks.  Risk  management 
requires  a  systematic  approach  to  problem  solving.  Program  managers  are  responsible 
for  ensuring  risk  management  programs  are  effective  for  DoD  systems  acquisitions.  By 
identifying,  analyzing,  and  managing  risks,  program  managers  can  have  a  significant 
impact  on  program  cost,  schedule,  and  performance.  The  March  15,  1996,  updates  to 
DoD  Directive  5000.1  and  DoD  Instruction  50(X).2  have  significantly  increased 
emphasis  on  the  importance  of  risk  management  in  managing  Defense  acquisition 
systems. 

Objectives.  The  audit  objective  was  to  evaluate  the  effectiveness  of  risk  management 
programs  for  Defense  acquisition  systems.  Specifically,  we  determined  whether  DoD 
risk  management  policies  and  procedures  for  Defense  acquisition  systems  were 
effectively  implemented  and  what  impact  risk  management  programs  had  on  reducing 
program  risks  and  costs.  We  also  reviewed  management  controls  as  they  applied  to  the 
audit  objectives. 

Audit  Results.  Risk  management  plans  developed  and  implemented  for  the  five 
Defense  acquisition  systems  reviewed  were  incomplete  and  noncompliant  with  the  DoD 
5000  series  of  documents.  As  a  result,  program  managers  were  not  using  risk 
management  plans  to  systematically  manage  and  reduce  program  cost,  schedule,  and 
performance  risks.  The  DoD  should  make  better  use  of  risk  management  software 
available  from  both  commercial  and  Government  sources. 

The  recommendation  in  this  report,  if  implemented,  will  assist  program  managers  in 
developing  valid  and  effective  risk  abatement  plans.  Through  risk  management  plans 
and  analjdiail  tools,  program  managers  can  provide  for  continuous  risk  assessments  to 
most  effectively  manage  identified  program  cost,  schedule,  and  performance  risks. 

Summary  of  Recommendation.  We  recommend  that  the  Under  Secretary  of  Defense 
for  Acquisition  and  Technology  task  the  Defense  Systems  Management  College  to 
systematically  review  and  catalogue  available  commercial  and  Government  risk 
management  systems'  applications  and  analytical  tools  in  the  Defense  Acquisition 
Deskbook  for  DoD  program  managers  to  adapt  and  use. 

Management  Comments.  We  received  comments  on  a  draft  of  this  report  from  the 
Director,  Test,  Systems  Engineering,  and  Evaluation.  He  concurred  with  the  intent  of 
the  recommendation  and  commented  on  statements  in  the  Finding.  Part  I  contains  a 
summary  of  the  management  comments  and  Part  HI  contains  the  complete  text  of 
management  comments. 
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Part  I  -  Audit  Results 


Audit  Results 


Audit  Background 


All  acquisition  programs  from  inception  to  completion  are  subject  to  risks.  Due 
to  the  complexity  of  current  weapon  systems,  risk  management  requires  a 
systenwtic  approach  to  problem  solving.  Program  managers  for  Defense 
acquisition  systems  are  responsible  for  ensuring  risk  management  programs  are 
effective.  By  identifying,  analyzing,  and  managing  riste,  they  can  have  a 
significant  impact  on  program  cost,  schedule,  and  performance.  According  to 
the  March  15,  1996  update  of  the  DoD  5000  series  of  directives,  program 
m^gers  are  to  use  risk  management  programs  to  identify  and  track  risk 
drivers,  define  risk  abatement  plans,  and  provide  continuous  risk  assessment 
through  each  acquisition  phase  to  determine  how  risks  have  changed.  Program 
managers  are  also  encouraged  to  accept  larger  risks  by  maximiying  return  on 
investments  through  trade-off  decisions  effecting  program  cost,  schedule,  and 
performance. 

The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  prescribes 
over^l  policy  for  acquisition  risk  management  programs  The  milestone 
decision  authority  for  each  major  Defense  system  is  responsible  for  overseeing 
the  overall  management  of  the  acquisitions,  including  implementation  of  risk 
management  programs. 


Audit  Objectives 


The  primary  audit  objective  was  to  evaluate  the  effectiveness  of  risk 
management  programs  for  Defense  acquisition  systems.  Specifically,  we 
determined  whether  DoD  risk  management  policies  and  procedures  for  Defense 
acquisition  systems  were  effectively  implemented  and  what  impact  risk 
management  programs  had  on  reducing  program  risks  and  costs.  We  also 
reviewed  management  controls  as  they  applied  to  the  audit  objectives.  See 
Appendix  A  for  the  audit  scope  and  methodology.  Prior  audit  coverage  related 
to  the  audit  objectives  is  in  Appendix  B.  Descriptions  of  the  five  Defense 
acquisition  systems  reviewed  are  in  Appendix  C. 
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Risk  Management  Plans 

Program  managers  for  the  five  Defense  acquisition  systems  reviewed  did 
not  implement  risk  management  plans  that  were  compliant  with  the 
DoD  5000  series  of  directives.  This  condition  was  caused  by  program 
managers  not  applying  the  analytical  tools  in  DoD  Manual  4245. 7-M, 
"Transition  from  Development  to  Production, "  for  identifying, 
assessing,  and  mitigating  risks  and  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  not  identifying  and  cataloging  risk 
management  systems'  applications  and  tools  for  program  managers  to 
adapt  and  use.  As  a  result,  program  managers  were  not  effectively  using 
risk  management  plans  to  systematically  manage  and  reduce  program 
cost,  schedule,  and  performance  risks. 


Background 


Risk  management  plans  are  plans  of  action  to  reduce  or  eliminate  risks  affecting 
program  cost,  schedule,  and  performance  for  Defense  acquisition  systems. 
Before  March  15,  1996,  DoD  Directive  5000.1,  part  1,  "Defense  Acquisition," 
February  23,  1991,  and  DoD  Instruction  5000.2,  part  5,  section  B,  "Defense 
Acquisition  Management  Policies  and  Procedures,"  February  23,  1991, 
provided  risk  management  guidance  for  Defense  program  managers.  Changes 
reflected  in  the  March  15,  1996,  update  of  the  DoD  5000  series  of  documents 
increased  the  emphasis  on  the  importance  of  risk  management.  The  changes 
reflect  the  Department's  acquisition  reform  efforts  to  empower  acquisition 
personnel  to  manage  risks  through  continuous  program  risk  assessment  rather 
than  to  avoid  risks. 

l^D  Directive  5000.1.  DoD  Directive  5000.1  requires  that  program  risks  and 
risk  management  plans  be  explicitly  assessed  at  each  milestone  decision  point 
before  approval  is  granted  to  proceed  into  the  next  acquisition  phase.  Risks 
identified  to  threat,  technology,  design  and  engineering,  support, 
manufacturing,  cost,  schedule,  and  risks  inherent  with  concurrent  program 
development  can  effect  program  results. 

DoD  Instruction  5000.2.  DoD  Instruction  5000.2  required  program  managers 
for  Defense  acquisition  systems  to  establish  risk  management  programs. 
Program  managers,  with  industty  ^d  user  participation,  were  to  establish  risk 
management  programs  that  will  identify  and  control  critical  risk  functions 
effecting  program  cost,  schedule,  and  performance  to  acceptable  levels.  With 
the  aid  of  technical  performance  "templates"  such  as  those  identified  in  DoD 
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Risk  Management  Plans 


Manual  4245. 7-M,  "Transition  from  Development  to  Production" 
^ptember  1985,  program  managers  were  to  develop  risk  management  programs 
mat  offer  nsk  abatement  techmques  to  support  sound  program  management 
decisions.  Specifically,  risk  management  programs  are  to: 

^®^^^bire  wd  document  risk  assessment  and  analysis  process,  with 
user  participation,  to  identify  risks  early  in  the  program; 

o  clearly  define  criteria  for  acquisition  activities  leading  to  the  risk 
assessment  events,  such  as  the  preliminary  and  the  critical  design  reviews; 

o  identify  and  track  risk  drivers,  define  risk  abatement  plans,  and 
provide  continuous  risk  assessment  throughout  each  acquisition  phase  to 
determine  how  risks  have  changed;  and 

0  define  evaluation  criteria  for  assessing  high,  moderate,  and  low 
ratings  of  risk  associated  with  each  subsystem  and  the  overall  system 

The  March  15,  1996,  update  of  the  5000  series  of  documents  did  not  change  the 
nsk  management  guidance. 

DoD  Manual  4245. 7-M.  DoD  Manual  4245. 7-M  provides  program  managers 
an  analytical  tool  for  developing  risk  abatement  plans.  The  manual  provides 
program  m^gers  with  templates  for  analyzing  potential  program  and  technical 
risls  associated  with  eight  critical  acquisition  functions:  funding,  design 
testmg,  production,  transition  planning,  facilities  and  capital  investment, 
logistics,  and  management.  Depending  on  the  critical  acquisition  function,  the 
Mwual  provides  program  managers  from  1  to  14  templates  to  identify  areas  of 
risk,^  outlme  ^k  abatement  plans,  and  identify  timelines  for  critical  path 
acquisition  activities.  Appendix  D  lists  the  risk  assessment  templates  provided 
for  the  eight  critical  acquisition  functions.  For  example,  the  Manual  provides 
SIX  nsk  assessment  templates  for  the  logistics  critical  acquisition  function.  The 
templates  address  the  acquisition's  logistics  support  analysis,  maimower  and 
personnel,  support  and  test  equipment,  training  and  material  equipment,  spares, 
and  techmcal  manuals.  r  . 

DoD  Manu^  4245.7-M  risk  management  guidance  with  its  risk  management 
templates  will  be  included  in  the  Defense  Acquisition  Deskbook  for  use  bv 
program  managers. 


Risk  Management  Plans  Developed 


Wsk  management  plans  were  developed  for  the  Combat  Service  Support  Control 
Syrtem  ^SSCS),  Fixed  Distributed  System  (FDS),  Hunter  Unmanned  Aerial 
Vehicle-Short  Range  Program  (Hunter  Program),  Minuteman  III  Guidance 
Replacement  Program  (GRP),  and  Single  Channel  Anti-Jam  Man-Portable 
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Risk  Management  Plans 


Tenninal  (SCAMP)  acquisition  systems.  However,  those  plans  did  not 
adeqiutely  address  risks  associated  with  the  eight  critical  acquisition  functions 
identified  in  DoD  Manual  4245.7-M.  In  addition,  those  plans  were  not  linked 
to  project  cost,  schedule,  and  performance  measurement  systems  for  tracking, 
adjusting,  and  validating  risk  abatement  plans.  Program  managers  and  prime 
contractors  developed  plans  that  also  understated  risks  by  overestimating 
expected  residts  from  risk  abatement  plans  and  failing  to  recognize  development 
and  integration  risks  associated  with  the  use  of  nondevelopmental  items. 
Further,  plans  were  not  continually  updated  as  systems  progressed  through 
acquisition  phases. 

Critical  Acquisition  Functions  Identified  in  DoD  Manual  4245.7-M.  Not  all 
critical  ac^isition  functions  identified  in  DoD  Manual  4245.7-M  were  folly 
examined  in  the  risk  management  plans  because  the  plans  were  not  collaborative 
efforts  between  program  offices,  prime  contractors,  and  users.  Specifically,  the 
SCAMP  and  Hunter  Program  Offices  developed  their  risk  management  plans 
without  contractor  participation.  The  prime  contractors  for  the  CSSCS,  FDS, 
and  GRP  Programs  developed  the  risk  management  plans  without  program 
office  participation.  Furflier,  none  of  the  plans  indicated  user  involvement  in 
the  plan  preparation. 

Program  Office  Plans.  The  Hunter  Program  Office  identified  and 
assessed  program  risks  using  the  risk  templates  for  the  eight  critical  acquisition 
functions  in  DoD  Manual  4245.7-M.  The  SCAMP  Program  Office  also 
identified  and  assessed  program  risks  but  did  not  apply  those  templates.  The 
reasonableness  of  SCAMP  Program  Office's  risk  management  assessments, 
however,  could  not  be  evaluated  because  documentation  did  not  exist  to  support 
its  assessments. 

Contractor  Plans.  With  the  exception  of  the  FDS  system  acquisition, 
only  design  or  production  acquisition  function  risks  associated  with  contractual 
line  item  deliverables  were  addressed  in  contractor-developed  risk  management 
plans.  The  prime  contractor  for  the  FDS  Program  Office  identified  and 
assessed  program  risks  using  those  templates  for  all  eight  critical  acquisition 
functions,  "nie  GRP  prime  contractor  only  addressed  one  of  the  nine  templates 
(manufacturing  plan)  provided  for  the  production  critical  acquisition  function  in 
preparing  its  risk  management  plan,  "nie  CSSCS  prime  contractor  addressed 
three  of  the^  fourteen  templates  (design  process,  design  requirements  and 
software  desi^)  provided  for  the  design  critical  acquisition  function  in 
preparing  its  risk  management  plan. 

Appendix  E  pnmarizes  the  extent  that  critical  acquisition  functions  were 
addressed  in  risk  management  plans  developed  for  the  five  Defense  acquisition 
systems  reviewed. 
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Applying  Template  Results.  If  properly  applied,  the  risk  templates  identified 
in  DoD  Manual  4245. 7-M  provide  a  sound  basis  for  assessing  overall  program 
risks  for  the  critical  functions  in  the  acquisition  process.  However,  results  from 
template  risk  assessments  were  not  lii^ed  to  the  program's  and  contractor's 
work  bre^down  structoes.  Work  breakdown  structures  define  project 
configmations  by  combining  hardware,  software,  services,  and  data  into 
components,  subassemblies,  and  assemblies.  To  identify  and  track  the  effect  of 
risk  abatement  plans  on  risk  drivers,  links  to  the  program  work  breakdown 
structures  are  needed  to  determine  the  effectiveness  of  risk  management 
abatement  plans  on  program  cost,  schedule,  and  performance. 

Risk  Assessments.  Program  managers  and  contractors  estimated  program  risks 
as  low  when  risk  abatement  plans  were  identified  and  when  nondevelopmental 
items  were  used  in  the  system  acquisition  strategies.  While  we  agree  diat 
program  risks  are  usually  lower  when  risk  abatement  plans  exist  to  reduce  or 
avoid  potential  problems,  risks  only  disappear  after  the  effectiveness  of  risk 
abatement  plans  have  been  demonstrated.  Further,  the  use  of  nondevelopmental 
items  often  present  system  integration  challenges  due  to  configuration 
modiftcations  and  changed  operating  environments,  which  can  signiftcantly 
increase  the  risk  of  meeting  cost,  schedule,  and  even  performance  requirements. 

Updating  Risk  Management  Plans.  None  of  the  risk  management  plans 
developed  for  the  five  acquisition  programs  were  reviewed  and  updated  during 
milestone  phases.  By  not  updating  the  risk  management  plans,  program 
managers  were  not  using  the  plans  to  track  the  effectiveness  of  identified  risk 
abatement  and  avoidance  efforts  and  to  implement  other  risk  abatement  plans 
for  newly  identified  program  cost,  schedule,  and  performance  risks. 


Effectiveness  of  Risk  Management  Programs 


The  effectiveness  of  risk  management  programs  on  reducing  program  risk  anfl 
costs  could  not  be  determined  for  the  five  Defense  acquisition  systems 
reviewed.  The  program  managers  did  not  obtain  data  base  systems'  applications 
to  systematically  measure  results  from  risk  abatement  plans  and  did  not  use  ^ta 
base  analytical  tools  to  test  and  validate  risk  abatement  plans  as  their  programs 
evolved. 

Beyond  reacting  to  results  from  cost,  schedule,  and  performance  reports,  only 
the  CSSCS  program  office  used  data  base  systems'  applications  to  extract, 
track,  and  modify  identified  risk  abatement  plans.  Further,  without  knowing 
full  ranges  of  consequences  by  testing  and  evaluating  plausible  assumptions, 
program  managers  had  more  difficulty  identifying  opportunities  to  avoid  and 
reduce  potential  problems. 
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Data  Base  Systems'  Applications  and  Analytical  Tools 


Data  base  systems'  applications  and  analytical  tools  exist  to  track,  adjust,  and 
validate  program  risk  abatement  plans.  Through  these  applications  and  tools, 
program  managers  can  link  the  DoD  Manual  4245. 7-M  templates  or  similar  risk 
management  aj^ytical  tools  with  the  update  of  the  5000  series  of  documents 
requirements  for  controlling  cost,  schedule,  and  performance  risks.  Although 
their  value  to  program  success  depends  on  die  quality  of  the  information  placed 
in  them,  analytical  tools  also  can  be  applied  with  data  base  systems'  applications 
to  manage  and  validate  program  risk  abatement  plans. 

Data  Base  Systems'  Applications.  Several  commercial  vendors  now  market 
project  management  data  base  systems'  applications  that  program  managers  can 
use  to  manage  program  risks.  Applied  successfully  by  numerous  public 
utilities,  research  and  development  facilities.  Defense  contractors,  and  other 
Government  and  industrial  entities,  the  data  base  applications  use  work 
breakdown  structures  to  extract,  track,  project,  and  m^ify  identified  risk- 
related  work  packages.  In  addition,  project  baselines  can  be  adjusted  if 
requirements  and  risk  abatement  assumptions  change. 

Analytical  Tools.  The  Defense  Systems  Management  College  has  developed 
the  "Performance  Analyzer,"  an  analytical  tool  for  analyzing  contractor 
performance  measurement  data.  The  Performance  Analyzer  software  compares 
acmal  program  results  with  planned  work  structure  breakdown  packages  and 
forecasts  expected  completions  for  cost  and  schedule.  Several  commercial 
vendors  also  have  developed  analytical  tools  for  use  in  risk  management.  The 
analytical  tools  allow  users  to  tailor  and  validate  their  risk  abatement  plans.  By 
testing  various  risk  abatement  assumptions,  program  managers  can  forecast 
ranges  of  e3q)ected  results  for  program  cost,  scheditie,  and  performance. 


Cost  and  Supportability  Goals 


The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  established  the 
Cost  as  an  Independent  Variable  Worldng  Group  to  address  approaches  and 
measures  to  reduce  life-cycle  costs.  The  Working  Group  recommended  that 
costs,  schedules,  and  performances  for  acquisition  systems  be  equally  weighted. 
Recognizing  that  die  DoD  has  tradition^y  managed  performance  risks,  the 
Working  Group  realized  that  managing  toward  cost  and  supportability  goals  will 
require  program  managers  to  make  trade-off  determinations  by  managing  risks 
to  produce  systems  that  fully  provide  military  capabiliQr.  As  a  result,  tiie  Under 
Secretary  tasked  flie  Director,  Test,  Systems  Engineering  and  Evduation,  in 
December  1995  to  review  management  practices  and  techniques  to  determine 
whether  new  approaches  were  needed  to  improve  risk  management  for  Defense 
acquisition  systems.  In  March  1996,  a  working  group  was  formed  to  review 
risk  management  practices  and  techniques. 
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Conclusion 


The  DoD  acquisition  community  needs  to  improve  its  efforts  to  develop  and 
implement  effective  risk  management  programs  to  manage  cost,  schedule,  and 
performance  risks.  This  area  is  of  continuing  concern  as  the  General 
Accounting  Office  initially  reported  this  condition  on  die  quality  of  risk 
management  programs  in  GAO  Report  No.  P]^lD-865  (OSD  Case 
No.  535034),  "Technical  Risk  Assessment:  the  Status  of  Current  DoD  Efforts," 
April  3,  1986.  As  recognized  by  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology  and  identified  by  prior  audits  and  reviews,  the 
quality  of  risk  management  programs  is  a  recurring  issue  for  management  of 
Defense  acquisition  systems.  To  fully  implement  the  Department's  acquisition 
reform  initiatives  and  comply  with  the  emphasis  placed  on  risk  identification 
and  management,  program  managers  must  improve  their  use  of  risk 
management  methodologies,  techniques,  and  tools.  Commercial  off-the-shelf 
data  base  systems'  applications  and  analytical  tools  exist  to  assist  program 
mangers  in  implementing  risk  management  programs.  Because  the  newly 
revised  DoD  acquisition  guidance  emphasizes  the  need  for  program  managers  to 
accept  larger  risks  to  achieve  breal^oughs  in  cost  and  performance,  these 
off-the-shelf  system  resoles  need  to  be  systematically  reviewed  and 
catalogued  in  die  Acquisition  Deskbook  for  DoD  program  managers  to  adapt 
and  use. 


Recommendation,  Management  Comments,  and  Audit 
Response 

We  recommend  that  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  task  the  Defense  Systems  Mans^ement  College  to  systematically 
review  and  catalogue  available  o^-the-shelf  commercial  and  Government- 
developed  risk  management  systems'  applications  and  tools  in  the  Defense 
Acquisition  Deskbook  for  DoD  program  managers  to  adapt  and  use. 

Director,  Test,  Systems  Engineering,  and  Evaluation,  Comments.  The 
Director,  Test,  Systems  Engineering,  and  Evaluation,  concurred  with  the  intent 
of  the  recommendation.  As  mentioned  in  die  finding,  he  stated  that  he  had 
assembled  a  Working  Group  to  review  the  current  risk  management  practices 
and  techmques  to  determine  whether  new  approaches  are  needed  to  improve  risk 
management.  At  the  completion  of  the  review,  he  stated  that  the  DoD  5000 
series  of  documents,  including  the  Defense  Acquisition  Deskbook,  the  Defense 
Acquisition  University  curriculum,  and  other  Service  acquisition  course 
materials,  would  be  updated  to  reflect  the  Department's  latest  policy  and 
procedures.  The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  is 
to  be  provided  the  preliminary  results  of  the  review  by  June  30,  1996. 
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Risk  Management  Plans 


We  also  received  management  comments  on  the  finding  from  the  offices  of  the 
Hunter  Unmanned  Aerial  Vehicle  Short  Range  Program  and  Minuteman  m 
Guidance  Replacement  Program.  Although  not  included  in  this  report,  we 
modified  the  finding,  as  appropriate,  in  response  to  the  management  comments 
received. 

Audit  Response.  The  comments  of  the  Director,  Test,  Systems  Engineering, 
and  Evaluation,  are  responsive  to  the  intent  of  the  audit  recommendation.  Ttie 
complete  text  of  management  comments  is  in  Part  HI. 
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Part  II  -  Additional  Information 


« 


Appendix  A.  Scope  and  Methodology 

Scope 


We  performed  this  program  results  audit  from  August  1995  through  February 
1996  in  accordance  with  auditing  standards  issued  by  the  Comptroller  General 
of  the  United  States  as  implemented  by  the  Inspector  General,  DoD. 
Accordingly,  we  included  tests  of  management  controls  as  considered  necessary. 
We  reviewed  acquisition  data  from  April  1989  through  January  1996  to 
accomplish  our  audit  objectives.  We  reviewed  the  effectiveness  of  risk 
management  strategies  on  reducing  program  risks  and  costs.  To  accomplish  this 
objective,  we  visited  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology,  the  Defense  System  Management  College,  and  the 
program  offices  and  contractors  for  die  programs  reviewed.  Appendix  F 
provides  a  complete  listing  of  the  organizations  visited  or  contacted. 

We  selected  five  Defense  acquisition  systems  for  review  to  evaluate  the 
effectiveness  of  their  risk  management  programs  for  managing  and  reducing 
program  cost,  schedule,  and  performance  risks.  The  acquisitions  reviewed 
were: 


0  DoD: 
o  the  Army: 


o  the  Navy: 

0  the  Air  Force: 


Hunter  Unmanned  Aerial  Vehicle-Short  Range 
Program 

Single  Channel  Anti-Jam  Man  Portable  Terminal 
and 

Combat  Service  Support  Control  System 

Fixed  Distributed  System 

Minuteman  HI  Guidance  Replacement  Program 


Methodology 


We  reviewed  the  program  offices'  risk  management  plans  and  programs, 
acquisition  strategy  reports,  cost  estimates,  cost  and  operational  effectiveness 
analyses.  Cost  Analysis  Improvement  Group  reports.  Defense  acquisition 
executive  summaries,  integrated  program  summaries,  milestone  acquisition 
decision  memorandums,  program  deviation  reports,  cost  performance  reports, 
test  and  evaluation  master  plans,  source  selection  data,  and  other  documents  as 
deemed  appropriate. 
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Appendix  A.  Scope  and  Methodology 


We  also  obtained  technical  direction  concerning  risk  management  program 
requirements  from  personnel  at  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition  and  Technology,  the  Defense  System  Management  College,  and  the 
Defense  Science  Board.  We  did  not  apply  statistical  sampling  or  rely  on 
computer-processed  data  to  support  the  finding  and  recommendation  in  this 
audit  report. 


Management  Control  Program 


Requirement  for  Management  Control  Review.  DoD  Directive  5010.38, 
"Internal  Management  Control  Program,"  April  14,  1987,  requires  DoD 
managers  to  implement  a  comprehensive  system  of  management  controls  that 
provides  reasonable  assurance  that  programs  are  operating  as  intended  and  to 
evaluate  the  adequacy  of  those  controls. 

Scope  of  Review  of  Management  Control  Program.  We  limited  our  review 
because  of  relevant  coverage  in  Inspector  General,  DoD,  Report  No.  96-028, 
"Implementation  of  the  DoD  Management  Control  Program  for  Major  Defense 
Acquisition  Programs,"  November  28,  1995.  The  report  discussed  the 
effectiveness  of  die  management  control  program  that  the  Defense  Acquisition 
Executive  and  the  Component  Acquisition  Executives  used  for  major  Defense 
acquisition  programs.  The  report  concluded  that  the  acquisition  community  had 
not  effectively  integrated  DoD  Management  Control  Program  requirements  into 
its  management  assessment  and  reporting  processes.  As  a  result  of  the  report 
recommendations,  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology  integrated  DoD  Directive  5010.38  requirements  into  the 
March  15,  1996,  revision  to  DoD  Directive  5000.1,  "Defense  Acquisition,"  and 
DoD  Regulation  5000. 2-R,  "Mandatory  Procedures  for  Major  Defense 
Acquisition  Programs  (MDAPS)  and  Major  Automated  Information  System 
(MAIS)  Acquisition  Programs. "  Acquisition  managers  are  now  to  use  program 
cost,  schedule,  and  performance  parameters  as  control  objectives  to  implement 
the  DoD  Directive  5010.38  requirements.  The  managers  are  to  identify 
material  weaknesses  through  deviations  from  approved  acquisition  program 
baselines  and  exit  criteria  in  the  Defense  Acquisition  Executive  Summary 
report.  Consequentiy,  we  limited  our  review  to  management  controls  over  the 
development  and  implementation  of  risk  management  programs  for  the 
five  Defense  acquisition  systems  reviewed. 

Adequacy  of  Management  Controls.  We  did  not  identify  any  material 
management  control  weaknesses  as  defined  by  DoD  Directive  5010.38. 
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Appendix  B.  Summary  of  Prior  Audits  and 
Other  Reviews 


During  the  past  5  years,  the  General  Accounting  Office  (GAO)  and  the  Office 
of  the  Inspector  General,  DoD,  issued  four  reports  that  discussed  aspects  of  risk 
management  programs  for  Defense  acquisition  systems.  In  addition,  a  1986 
GAO  report  discussed  the  quality  of  risk  assessments  that  program  offices  for 
Defense  acquisition  systems  made. 


General  Accounting  Office 


GAO  Report  No.  NSIAD-94-255BR  (Office  of  the  Secretary  of  Defense  [OSD] 
Case  No.  9785),  "1995  Defense  Budget;  Potential  Reductions  and  Rescissions 
in  RDT&E  [Research,  Development,  Testing,  and  Evaluation]  and  Procurement 
Programs,"  September  8,  1994,  examined  the  DoD  FY  1995  budget  request  and 
prior  years'  appropriations  for  selected  programs.  The  GAO  suggested 
deferring  the  Air  Force  Joint  Primary  Aircraft  Training  System  production  until 
the  design  was  more  firm  to  reduce  program  risk  while  still  allowing 
development  and  testing  as  scheduled.  The  report  made  no  recommendations. 

GAO  Report  No.  NSIAD-91-280  (OSD  Case  No.  8733),  "Tactical  Missile 
Acquisition:  Understated  Technical  Risk  Leading  to  Cost  and  Schedule 
Overruns,"  September  17,  1991,  recommended  that  DoD  ensure  the  Defense 
research  and  engineering  directorate  independently  review  program  office's 
technical  risk  assessments.  DoD  concurred  and  indicated  that  technical  risk 
assessments  were  institutionalized  within  the  DoD  and  were  an  integral  part  of 
milestone  documentation. 

GAO  Report  No.  IMTEC-91-030  (OSD  Case  No.  8640),  "Submarine  Combat 
System;  BSY-2  Development  Risks  Must  Be  Addressed  and  Production 
Schedule  Reassessed,"  August  22,  1991,  recommended  that  the  Navy  assess  the 
BSY-2  submarine  risks,  determine  &eir  impact,  adjust  the  development 
approach,  and  report  the  results  to  Congress.  The  Navy  did  not  concur  with  the 
recommendation,  but  stated  that  it  would  constantly  monitor  the  risk  areas  to 
ensure  the  BSY-2  system  meets  baseline  thresholds. 
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Appendix  B.  Summary  of  Prior  Audits  and  Other  Reviews 


GAO  Report  No.  PEMD-865  (OSD  Case  No.  535034),  "Technical  Risk 
Assessment:  the  Status  of  Current  DoD  Efforts,"  April  3,  1986,  found  DoD  did 
not  clearly  define  technical  risk  and  had  insufficient  direction  for  various 
ai^^ical  approaches.  GAO  also  found  technical  risk  programs  had  not  met 
minimal  standards  in  most  of  25  program  offices  reviewed.  In  addition, 
essential  information  on  assessment  procedures  and  results  was  often  not 
available  to  program  managers  or  reviewers.  The  report  recommended  that  the 
Secretary  of  Defense: 

0  deEne  technical  risk  and  categories  for  rating  risk; 

o  require  risk  efforts  to  focus  on  tecWcal  risk  and  be  repeated  early 
and  late  in  each  acquisition  phase; 

0  require  program  offices  to  document  risk  assessment  procedures  and 

results; 

o  establish  guidelines  for  rating  risks,  scope,  data  collection,  and 
assessment; 

o  require  technical  risk  information  to  include  format,  scope,  data 
collection,  sources  of  risk  information,  and  assessment  approaches;  and 

0  provide  more  focused  training  in  technical  risk  assessment. 

DoD  concurred  fully  or  partially  with  all  recommendations  except  the 
recommendation  pertaining  to  making  risk  assessment  procedures  available  for 
review.  Our  current  audit  showed  that  DoD  had  yet  to  folly  correct  the 
conditions  identified  in  this  GAO  report. 


Inspector  General,  DoD 


Inspector  General  Report  No.  94-101,  "Program  Management  Organization  for 
the  Upper  Tier  Theater  Missile  Defense  System,"  May  16,  1994,  stated  that 
DoD  and  contractor  organizations  could  be  more  effectively  organized  to  reduce 
program  developmental  and  systems  integration  risk  for  the  Theater  High 
Altitude  Area  Defense  missile  system  and  the  ground-based  radar.  The  report 
recommended  a  merger  of  the  Theater  High  Altitude  Area  Defense  and  Ground- 
Based  Radar  Project  Offices  and  an  Engineering  and  Mannfartiiring 
Development  contract  for  a  single  prime  contractor.  Management  concurred 
with  the  recommendations.  The  Theater  High  Altitude  Area  Defense  and 
Ground-Based  Radar  Project  Offices  became  a  single  project  office  effective 
June  30,  1995. 
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Appendix  C.  Programs  Reviewed 


Combat  Serrice  Support  Control  System  Program.  The  Army  manages  the 
CSSCS  program.  CSSCS  is  a  computer  software  system  designed  to  assist 
commanders  and  their  staffs  in  processing  and  executing  large  volumes  of 
logistical,  personnel,  and  medical  information.  The  system  consists  of 
transportable  and  lightweight  computer  imits,  common  Army  operating 
environment  software,  and  CSSCS-unique  software.  The  Army  structured  the 
software  development  to  evolve  incrementally  over  five  versions,  each  building 
on  the  capabilities  of  the  predecessor.  TRW,  Data  Technologies  Division,  is 
the  prime  contractor  for  fliis  systems  acquisition.  Production  and  fielding  of  the 
first  foil  system  is  scheduled  for  FY  1998.  The  RDT&E  and  procurement  cost 
baseline  for  the  CSSCS  is  $263.8  million  (then-year  dollars). 

Fixed  Distributed  System  Program.  The  Navy  manages  the  FDS  program. 
FDS  is  a  passive  surveillance  system  for  detecting  submarines.  Using 
hydrophones  placed  on  the  sea  floor,  the  FDS  transmits  acoustic  data  to  a  shore 
station  for  processing  and  analyzing.  American  Telephone  and  Telegraph 
Federal  Systems  Advanced  Technology  and  Loral  Federal  Systems  are  the  prime 
contractors  for  this  systems  acquisition. 

The  program's  Engineering  and  Manufacturing  Development  acquisition  phase 
was  90  percent  complete  and  is  expected  to  conclude  in  September  1996. 
Production  of  the  FDS  has  been  cancelled  due  to  a  change  in  threat  Total 
acquisition  cost  was  proposed  in  excess  of  $9  billion  (then-year  dollars),  but  has 
been  reduced  to  $1.2  billion  (then-year  dollars)  as  a  result  of  production 
cancellation. 

Hunter  Unmanned  Aerial  Vehicle-Short  Range  Program.  The  Army 
executes  the  DoD  joint-managed  Hunter  Program.  The  Hunter  is  a  short  range, 
small  fixed-wing  aircraft  remotely  piloted  from  a  groimd  control  station.  The 
Hunter's  primary  mission  is  to  relay  near-real-time  video  telemetry  information 
to  battlefield  conunanders  from  target  areas  up  to  150  kilometers  beyond  the 
forward  line  of  our  own  troops.  The  Hunter  provides  battlefield  commanders 
reconnaissance,  surveillance,  and  target  acquisition  intelligence  an  average  of 
16  hours  for  every  24  hours.  TRW,  Avionics  and  Surveillance  Group,  is  the 
prime  contractor. 

The  Under  Secretary  of  Defense  for  Acquisition  and  Technology  cancelled  the 
$2.5  billion  (RDT&E,  Procurement  and  Military  Construction,  then-year 
dollars)  program  on  January  31,  1996.  Before  cancellation,  .seven  systems  were 
acquired  at  a  cost  of  $169.7  million. 
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Appendix  C.  Programs  Reviewed 


Minuteman  EDI  Guidance  Replacement  Program.  The  Air  Force  manages  the 
GRP  program.  The  GRP  replaces  guidance  systems  for  the  Minuteman  HI 
missile  and  converts  the  software  from  "FORTRAN"  to  "Ada. "  This  upgrade 
will  extend  the  life  of  the  missile  beyond  FY  2020.  Also,  the  GRP  will 
improve  system  fault  detection  and  isolation  and  maintain  the  option  to 
configure  Ae  missile  with  a  different  re-entry  vehicle  and  an  advanced  inertial 
measmenaent  unit.  Rockwell  International  Autonetics  Strategic  Systems 
Division  is  the  prime  contractor  for  hardware  and  software  integration,  and 
Honeywell  Space  and  Strategic  Systems  is  die  primary  subcontractor  for 
computer  hardware  and  software.  The  RDT«feE  and  Procurement  cost  baseline 
for  the  GRP  is  $1.84  billion  (then-year  dollars). 

Single  Channel  Anti-Jam  Man-Portable  Terminal  Program.  The  Army 
manages  the  SCAMP  program.  SCAMP  is  a  hand-carried,  battery-powered 
extremely  high  frequency  satellite  communications  terminal  used  with  the 
Military  Strategic/Tactical  Relay  system.  Operation  of  the  terminal  enables 
commanders  in  chief  and  other  high  priority  users  to  transmit  secure  voice  and 
data  traffic  and  receive  command  and  control  voice  and  data  traffic  from  a  base 
station.  Lockheed  Missiles  and  Space  Company,  Space  Systems  Division,  and 
Martin  Marietta,  Communication  Systems,  were  each  awarded  a  cost-plus-lfeed- 
fee  prime  contract. 

The  Engineering  Manufacturing  Development  acquisition  phase  contracts  were 
terminated  (Lockheed  in  September  1993  and  Martin  Marietta  in  November 
1994)  due  to  escalating  cost  growth  and  program  funding  cuts.  Despite  contract 
terminations,  a  fixed-price,  full-scale  SCAMP  production  contract  will  be 
awarded  in  FY  1996.  The  RDT&E  and  Procurement  cost  baseline  for  the 
SCAMP  is  $220.2  million  (then-year  dollars). 
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Appendix  D.  List  of  Templates  for  Critical 
Acquisition  Functions 


Mfg.  Manufacturing 
Ref.  Reference 


18 


Appendix  E.  Summary  of  Critical  Acquisition 
Functions  Addressed  in  Risk  Management  Plans 


Acquisition 


Functions 

CSSCS 

FDS 

GRP 

Hunter 

SCAMP 

Design 

partially 

yes 

no 

yes 

partially 

Facilities 

no 

yes 

no 

partially 

no 

Funding 

no 

yes 

no 

yes 

no 

Logistics 

no 

yes 

no 

yes 

partially 

Management 

no 

yes 

no 

yes 

partially 

Production 

no 

yes 

partially 

partially 

no 

Test 

no 

yes 

no 

yes 

no 

Transition  Plan  no 

yes 

no 

yes 

no 
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Appendix  F.  Organizations  Visited  or  Contacted 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology,  Washington,  DC 
Acquisition  Program  Integration,  Washington,  DC 
Strategic  and  Tactical  Systems,  Washington,  DC 


Department  of  the  Army 

Assistant  Secretary  of  the  Army  (Research,  Development  and  Acquisition), 
Washington,  DC 

Program  Executive  Office,  Command,  Control,  and  Communications  Systems, 
Fort  Monmouth,  NJ 

Military  Strategic/Tactical  Relay  System  Program  Office,  Fort  Monmouth,  NJ 
Strategic  and  'Dieater  Command  and  Control  Systems  Program  Office, 

Fort  Belvoir,  VA 

Army  Audit  Agency,  Fort  Monmouth,  NJ 


Departments  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller), 
Washington,  DC 

Undersea  Surveillance  Program  Director,  Arlington,  VA 
Fixed  Distributed  System  Program  Office,  Arlin^on,  VA 
Program  Executive  Office,  Cruise  Missiles  and  Unmanned  Aerial  Vehicles, 
Arlington,  VA 

Hunter  Unmanned  Aerial  Vehicle  -  Short  Range  Program  Office,  Huntsville,  AL 


Departments  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller), 
Washington,  DC 

Program  Executive  Office  for  Bombers,  Missiles  and  Trainers,  Washington,  DC 
Silo  Based  Intercontinental  Ballistic  Missile  System  Program  Office, 

Hill  Air  Force  Base,  UT 
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Appendix  F.  Organizations  Visited  or  Contacted 


Other  Defense  Organizations 

Director,  Defense  Systems  Management  College,  Fort  Belvoir,  VA 


Contractors 

American  Telephone  and  Telegraph  Advanced  Technology  Systems,  Greensboro,  NC 
Loral  Federal  Systems,  Manassas,  VA 
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Appendix  G.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  for  Acquisition  and  Technology 
Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 
Commandant,  Defense  Systems  Management  College 
Director,  Defense  Logistics  Studies  Information  Exchange 
Director,  Test,  Systems  Engineering,  and  Evaluation 
President,  Defense  Acquisition  University 
Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Assistant  to  the  Secretary  of  Defense  ^blic  Affairs) 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 

Program  Executive  Office  Command,  Control,  and  Communications  Systems 
Program  Manager,  Military  Strategic/Tactical  Relay  System 
Program  Manager,  Strategic  and  Tlieater  Command  and  Control  Systems 


Department  of  the  Navy 

Assistant  Secret^  of  the  Navy  (Financial  Management  and  Comptroller) 
Program  Executive  Office  for  Cruise  Missiles  and  Unmanned  Aerial  Vehicles 
Program  Manager,  Hunter  Unmanned  Aerial  Vehicle-Short  Range 
Commander,  Space  and  Naval  Warfare  Systems  Command 
Program  Manager,  Undersea  Surveillance 
Program  Manager,  Fixed  Distributed  System 
Auditor  General,  Department  of  the  Navy 

Department  of  the  Navy,  Dudley  Knox  Library,  Naval  Postgraduate  School 


Department  of  the  Air  Force 

Assistant  Secret^  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Program  Executive  Office  for  Bombers,  Missiles  and  Trainers 

Program  Manager,  Silo  Based  Intercontinental  Ballistic  Missile  System 
Auditor  General,  Department  of  the  Air  Force 
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Appendix  G.  Report  Distribution 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 


Non-Defense  Federal  Organizations 

Office  of  Management  and  Budget 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 
General  Accounting  Office 

Chairman  and  ranking  minority  member  of  the  following  congressional  committees  and 
subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 
House  Committee  on  Government  Reform  and  Oversight 
House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal 
Justice,  Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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Part  ni  -  Management  Comments 


Director,  Test,  Systems  Engineering,  and 
Evaluation,  Comments 


OFRCE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON  DC  20301-3000 


0  4  JUN  1996 

MEMORANDUM  FOR  DEPARTMENT  OF  DEFENSE  INSPECTOR  GENERAL 

SUBJECT;  The  Department  of  Defense  Inspector  General  Audit 
Report  on  Risk  Management 


This  is  the  Under  Secretary  of  Defense  for  Acquisition  and 
Technology,  USD(A&T),  response  to  the  Department  of  Defense 
Inspector  General  (DoD-IG)  draft  report  SAE-OOe,  "Audit  Report  of 
Risk  Management  Programs  for  Defense  Acquisition  Systems.'  The 
USD(A&T)  partially  concurs  with  the  findings  of  the  DoD-IG 
report.  The  report  states  that  all  five  (5)  programs  audited: 
DoD's  Hunter  Unmanned  Aerial  Vehicle-Short  Range  Program  (Hunter 
Program) ;  the  Army's  Single  Channel  Anti- Jam  Man  Porteible 
Terminal  (SCAMP);  Combat  Service  Support  Control  System  (CSSCS) ; 
the  Navy's  Fixed  Distributed  System  (FDS) ;  and  the  Air  Force's 
Minuteman  III  Guidance  Replacement  Program  (GRP) ,  did  not 
implement  risk  management  plans  that  are  con^liant  with  the  DoD 
5000  series  documents.  The  audit  report  contends  that  this 
noncoit^liance  was  caused  by  program  managers  not  applying  the 
analytical  tools  in  DoD  Manual  4245. 7-M,  Transition  from 
Development  to  Production,  September  1995. 

We  agree  that  these  programs  did  not  all  apply  the 
analytical  tools  in  DoD  Manual  4245. 7-M  as  part  of  their  risk 
meuiagement  efforts.  However,  it  should  be  noted  that  the  DoD 
Manual  42 4 5. 7-M  was  only  intended  to  be  a  guide,  not  a  mandatory 
requirement,  for  programs  to  use  in  establishing  risk  management 
plains.  Further,  since  each  program  has  a  risk  management  plan  in 
place,  we  believe  they  are  compliant  with  DoD  5000  series 
documents . 

We  would  also  point  out  that  managing  risks  in  our  systems 
is  an  exhaustive  and  iterative  process.  No  set  of  tools  and 
software  application  packages  can  guarantee  an  effective  risk 
management  program.  Per  the  current  Acquisition  Reform 
Initiative,  all  programs,  new  and  old,  are  required  to  iirplement 
the  Integrated  Product  and  Process  Development  concept  in  all 
phases  of  their  acquisition  program.  This  initiative  includes 
putting  together  risk  management  plans  that  are  based  on  a  well 
integrated  approach  in  all  programs.  Integrated  risk  m^agement 
is  not  an  independent  and  separate  entity  from  acquisition 
programs.  It  is  an  integral  part  of  the  overall  acquisition 
program  management.  How  well  a  program  is  managed  to  meet  its 
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cost,  schedule  and  performance  requirements  is  indicative  of  how 
well  inherent  program  risks  are  managed  via  an  integrated 
approach. 

We  concur  with  the  intent  of  the  IG  recommendation  that 
USD(A&T)  take  steps  to  include  relevant  material  on  risk  in  the 
upcoming  Defense  Acquisition  deskbook  for  DoD  Program  Managers  to 
adapt  and  use.  However,  as  the  Office  of  Primary  Responsibility 
in  DoD,  I  have  developed  and  am  implementing  a  much  more 
ambitious  plan  to  address  concerns  very  similar  to  those  of  the 
IG.  Specifically,  at  the  direction  of  USD{A&T) ,  I  have  assembled 
a  Working  Group  with  the  Services,  DSMC,  DoD  agencies,  DoD-IG, 
industry,  and  key  Office  of  the  Secretary  of  Defense  staff  to 
review  the  cxirrent  risk  management  practices  and  techniques  and 
determine  whether  new  approaches  are  needed  to  inprove  risk 
management.  As  a  result,  the  DoD  5000  series  documents, 
including  the  desJcbook,  Defense  Acquisition  University  curriculum 
and  other  Service  acquisition  course  materials  will  be  updated  to 
reflect  the  Department's  latest  policy  and  procedures.  The 
preliminary  results  of  the  study  will  be  given  to  USD(A&T)  in 
June  1996. 

If  you  have  any  questions,  please  contact  my  action  officer, 
Mr.  Burhan  Adam,  (703)  681-4534,  (e-mail)  Adamby@acq.osd.mil. 


ZjStir/K.  Burt 

JDirector,  Test,  Systems 

Engineering,  and  Evaluation 
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